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ABSTRACT 

We extend the Figure of Merit formalism usually adopted to quantify the statistical 
performance of future dark energy probes to assess the robustness of a future mission 
to plausible systematic bias. We introduce a new robustness Figure of Merit which 
can be computed in the Fisher Matrix formalism given arbitrary systematic biases in 
the observable quantities. We argue that robustness to systematics is an important 
new quantity that should be taken into account when optimizing future surveys. We 
illustrate our formalism with toy examples, and apply it to future type la supernova 
(SNIa) and baryonic acoustic oscillation (BAO) surveys. For the simplified systematic 
biases that we consider, we find that SNIa arc a somewhat more robust probe of dark 
energy parameters than the BAO. We trace this back to a geometrical alignement 
of systematic bias direction with statistical degeneracy directions in the dark energy 
parameter space. 
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1 INTRODUCTION 

The discovery of the accelerating universe has been hailed 
as one of the most important developments in cosmology in 
decades. Yet the physical nature of the cause of this accelera- 
tion - the dark energy - is lacking. With that in mind, a large 
amount of effort has gone into measuring the parameters 
describing dark energy, notably its equation of state param- 
eter, w = p/p, where p and p are the dark energy pressure 
and energy density. In fact, measuring w to high accuracy 
is one of the most important goals of ongoing and upcom- 
ing large scale cosmological surveys, such as Dark Energy 
■ Survey (DEEQ), Baryon Oscillation Spectroscopic Survey 
; (BOS^I), Large Synoptic Survey Telescope (LSSlEI), S quare 
] Kilometre Array (SKAQ), Euclid (|Laureiis et al.ll2009^ . 

In order to rank proposed future dark energy missions 
according to their potential capabilities, a series of Figures 
of Merit (FoMs) has been introduced, whose aim is to quan- 
tify the science return of an experiment in terms of its abil- 
ity to constrain dark energy. Perhaps the most widely used 
FoM is t he one identified by the d ark energy task force 
(DETF) (|Albrecht et all |2006| . [ioogi ). which is a measure 
of the statistical power of a future dark energy mission. 
Other higher dimensional versions have also been consid- 



^ http:/ /www.darkcnergysurvey.org 
2 http://cosmology.lbl.gov/BOSS/ 
^ http://www.lsst.org 
^ http://www.skatclescope.org/ 



ered (iHuterer fc Turneil I2OOII: lAlbrecht fc Bernsteinl |2007| : 



IWanell2008l : ICrittenden et al.H2009l : iMortonson et aLlboiof ). 

The most general approach to performance forecasting in- 
volves the use of a suitably defined utility function in the 
Bayesian f ramework, and it has recently been presented in 
iTrotta et al., (,201Q ). 



The purpose of this paper is to expand the FoM formal- 
ism to consider a new dimension of the performance of a fu- 
ture dark energy probe, which has been until today largely 
neglected - namely, its robustness to potential systematic 
errors. It is well known that systematic errors are going to 
be one of the most challenging factors limiting the ultimate 
statistical performance of precision measurements of w{z). 
Yes there has been until now no formal way to quantify how 
prone to potential systematic a future dark energy measure- 
ment might be. This work takes a first step towards redress- 
ing this issue, by introducing a so-called "robustness" FoM 
which complements the statistical FoMs mainly considered 
so far in the literature. 



This paper is organized as follows: in section [5] we intro- 
duce our statistical and robustness FoMs, whose properties 
are illustrated in section [S] We then apply this formalism to 
future supernovae type la and baryonic acoustic oscillation 
data in section [4] Our results and conclusions are presented 
in section [5] 
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2 FIGURES OF MERIT FOR FUTURE DARK 
ENERGY PROBES 

2.1 Gaussian linear model 

Suppose we have 2 different dark energy probes, whose like- 
lihood function is assumed to be Gaussian and is character- 
ized by a Fisher matrix Li (i = 1, 2), i.e. 



C,{Q) = p{d,\Q) = ^exp --(/i, - e)'Li(/i, - e) 



the evidence for data d is given by 



(1) 

where are the parameters one is interested in constrain- 
ing and is the location of the maximum likelihood value 
in parameter space. In the absence of systematic errors, the 
maximum likelihood point, /li, is located at the true param- 
eters value, which can be taken to be the origin. We are 
here neglectic realization noise, i.e. we are working in the 
Fisher matrix framework, in which is interpreted as the 
expectation value of the maximum likelihood estimator av- 
eraged over many data realizations. However, the presence 
of unmodelled systematic errors would introduce a non-zero 
shift in Hi. Below, we will show how the systematic shifts ^i 
can be estimated by propagating onto the parameter space 
the shifts resulting from plausible unmodelled systematics 
in the observables for dark energy. 

The posterior distribution for the parameters, p{Q\D), 
is obtained by Bayes theorem as 



p{Q\D) = 



p(Q)p{D\Q) 
p{D) ' 



(2) 



where p{0) is the prior, p{D) the Bayesian evidence and 
D are the data being used. If we assume a Gaussian prior 
centered on the origin with Fisher matrix E, the posterior 
from each probe is also a Gaussian, with Fisher matrix 



F,^L, + E (i = 1, 2) 
and posterior mean 

Mi = Fi^^iLifJ-i). 



(3) 



(4) 



If we combine the two probes, we obtain a Gaussian poste- 
rior with Fisher matrix 



F = Li + La + E 



and mean 



(5) 



(6) 



Notice that the precision of the posterior (i.e., the Fisher 
matrix) does not depend on the degree of overlap of the 
likelihoods from the individual probes. This is a property of 
the Gaussian linear model. In the presence of systematics, a 
FoM based on the posterior Fisher matrix is thus insufficient 
to quantify the power of the experiment: a future probe sub- 
ject to systematic bias would have the same statistical FoM 
as an unbiased experiment. This motivates us to extend our 
considerations to a second dimension, namely robustness. 

For future reference, it is also useful to write down the 
general expression for the Bayesian evidence. For a Normal 
prior p{0) ~ N{9Tr, E) and a likelihood 



z;(e) = roexp(--(6(o 



e)'L(eo-e) 



(7) 



p(d) 



dep(d|e)p(e) = £(, 



exp 



(8) 



where F = L + T, a.nd 9 = F-^{L9o + T,9„) 



2.2 The Statistical Figure of Merit 

It has become customary to describe the statistical power 
of a future dark energy probe by the inverse area of its 
covariance matrix. This measure of st atistical performance 
- widely known as the DETF FoM (|Albrecht et all l2006l : 
iHuterer fc Turner! l200ll 'l - is usually defined (up to multi- 
plicative constants) as 



1/2 



(9) 



Here, we suggest to adopt a more statistically motivated 
measure of the information gain, namely the KuUback- 
Leibler divergence (KL) between the posterior and the prior, 
representing the information gain obtained when upgrading 
the prior to the posterior via Bayes theorem: 



Dkl = 



p(e|D)in^de. 
p(e) 



(10) 



The KL divergence measures the relative entropy between 
the two distributions: it is a dimensionless quantity which 
expressed the information gain obtained via the likelihood. 
For the Gaussian likelihood and prior introduced above, the 
information gain (w.r.t. the prior S) from the combination 
of both probes is given by 



Dkl = - (In \F\ - In iE| - tr[l - EF"']) 



(11) 



Below, we shall be interested in assessing the statisti- 
cal performance of future dark energy probes, in a context 
where probe 1 is taken to represent present-day constraints 
on dark energy parameters, while probe 2 is a future dark en- 
ergy mission. We normalize the KL divergence for the com- 
bination of probe 1 and probe 2, given by Eq. (|lip . w.r.t. the 
case where probe 2 is assumed to be a hypothetical experi- 
ment that would yield the same dark energy constraints as 
the existing ones (probe 1). This is not meant to represent 
a realistic dark energy probe, but merely to give a bench- 
mark scenario for the normalization of the information gain. 
This choice of normalization has the added advantage of 
cancelling out most of the prior dependence in Eq. pip . Af- 
ter exponentiating the normalized KL divergence, we there- 
fore suggest to adopt as a statistical FoM the dimensionless 
quantity 

„ |Li+L2 + E|i/2 



|2Li -f E|i/2 



exp Qtr[E((Li + La + E)"' - (2Li + E)"') 



2.3 Robustness of Dark Energy Probes 



(12) 



In order to quantify the robustness to potential systematics 
of a combination of probes, we wish to derive a measure of 
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the degree of consistency between them. The gist of our new 
robustness FoM is that our confidence in the robustness of a 
new dark energy probe is increased if it returns constraints 
which overlap significantly with previously existing probes. 
If on the contrary the new probe has a small degree of consis- 
tency with previous experiments, this might point to either 
a failure of the underlying theoretical model or to the pres- 
ence of unmodelled systematics in the new probe (or both) . 
In the following, we focus on the latter hypothesis. 

The idea is to perform a Bayesian model comparison 
between two hypotheses, namely Ho, stating that the data D 
are all compatible with each other and the model, versus Hi, 
purporting that the observables are incompatible and hence 
tend to pull the constraints in different regions of parameter 
space. The Bayes factor between the two hypotheses, giving 
the relative probabilities (odds) between Ho and Hi is given 
by 



R ■- 



p(.D\Ho) 



YltiPiM-HiY 

where the Bayesian evidence for a given hypothesis H is 



(13) 



p{d\H)= / A(dp{d\Q,H)p{Q\H). 



(14) 



If -R 3> 1, this is evidence in favour of the hypothesis Ho 
that the data are compatible. If instead R the alterna- 
tive hypothesis Hi is preferred (namely, that the data are 
incompatible). Examples of the appli cation of the s t atistic s 
R introduced above can be found in iHobson et al] (l2002h : 
iFeroz et all (|2008l ') - see the Appendix of iFeroz et al] (|2009l ') 
for a toy model illustration. For a review of Bayesian meth- 
ods in cos mology, and in particular of model selection tech- 
niques, see lTrottal (|2008l ). 

We can restrict our considerations to just two probes, 
hence D = {di,d2}. Then the criterium of Eq. (|13|l can be 
written as (omitting for brevity the explicit conditioning on 
hypotheses) 



R = 



p{di,d2) _ p{d2\di)p{di) _ p{d2\di) 



p{di)p{d2 



p{di)p{d2 



P{d2) 



(15) 



The conditional evidence for d2 given dataset di can be cal- 
culated as 



p{d2\di 



p{d2\Q)p{Q\di)dQ, 



(16) 



where the first term is the likelihood for the second probe 
and the second term is the posterior from the first probe. 
By using the likelihood (U]), and making use of Eq. ((8]) we 
obtain: 



p{d2\di 



<2)\Fl\ 



1/2 



|F|l/2 



exp 



(17) 



where fj. is given by Eq. (|6}, F by Eq. ([5]) and fii by 
Eq. (jl}. Using again Eq. (|8]) we obtain for the denomina- 
tor in Eq. JTHJ 



P(rf2)=A^P^exp 



-7; (Ai2^2M2 - ^J'lF2^i2) 



(18) 



so that we obtain 

|Fi|l/2|77',|l/2 



R 



exp 



Fll/2|Ell/2 
1 



t^{Fifi^ + M2-F'2M2 - f^^Ffi) 



(19) 



Therefore we can recast Eq. (|13|) into 



It 1 _t _ 



1, \F\ 

2'^m + 2 



(20) 
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We shall use below the robustness R to define a new FoM. 
For now, let us notice that it is the product of two terms: 
the terms involving determinants of the Fisher matrices add 
up to an Occam's razor factor, which is always > 0. The 
second part (summing over quadrating forms involving the 
various Fisher matrices) expresses the degree of overlap of 
the posteriors from the two probes. This term will reduce R 
if the posteriors from the two probes are significantly dis- 
placed from the posterior obtained using the combined data 
set (a smoking gun for systematic bias). The generalization 
of Eq. (|2Up to an arbitrary number of probes is derived in 
the Appendix. 

2.4 The Robustness Figure of Merit 

We now specialize to the situation where probe 1 describes 
our current knowledge about dark energy parameter, while 
probe 2 represents a proposed future dark energy mission. 
Notice that probe 1 does not need to be a single experiment 
(i.e., just SN la or just BAO), but it can be interpreted as 
being the effective joint constraint from a combination of 
all available present-day dark energy probes. Without loss 
of generality, we assume that the current constraints are 
unbiased, i.e. we set /ii = in the following, and we wish 
to evaluate the robustness of a future probe, as defined in 
Eq. (|20l) . which might be subject to systematic bias. 

Let us assume for the moment being that we can es- 
timate the bias b in parameter space which probe 2 might 
be subject to. A procedure to achieve this will be presented 
below for the specific cases of SN la and BAO observations. 
For now, we remain completely general, and assume that the 
maximum likelihood estimate for the dark energy parame- 
ters from probe 2 is displaced from their true value by a bias 
vector 6, i.e. IJ.2 ~ b. This, together with the assumption that 
probe 1 is unbiased (i.e., ^1 — 0) gives p,2 = F2^L2b and 
the joint posterior mean from both probes is 

yL = F-^L2b. (21) 

Then we can write for the robustness R, Eq. (|20p 

In J? = i(F~'L26)'F(F-'L26) - i(6L2)*f2"'(i2&) 



1, 1^1 



\Fi\ 



(22) 



which can be rewritten as 



InR^ -^{bL2)\F2^ - F-^){L2b) +Tlo 
= -^b'F*b + Ho 



(23) 
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where we have defined 



F' = L2{F-' - F-')L2 



7^o 



■In 



|S| 



(24) 
(25) 



If the prior E is neghgible with respect to L2 we have F2 = 
L2 and F* =F2- F2F"^F2. 

In order to normalize the value of R, we adopt the 're- 
peated experiment' procedure we used for the normalization 
of the statistical FoM. This is defined as the hypothetical 
case where the new experiment (probe 2) yields exactly the 
same Fisher matrix as the existing probe 1, and is unbiased, 
i.e. Fi = F2 and b = (0, 0). For this identically repeated case 
the robustness of the two probes is given by 



(26) 



(|2Li + E||S|)i/2- 

Normalizing K from Eq. H23p to the above value means 
that R/R* = 1 is the robustness that one would achieve by 
carrying out a new dark energy measurement that would 
yield exactly the same constraints as we currently have, and 
no bias. We therefore define the quantity 



Rn = ^ 



R_ 

R, 



(27) 



as our robustness FoM, which expresses the robustness of 
probe 2 under the assumption that it will be afi'ected by a 
bias b. 

The robustness FoM above represents a "worst case sce- 
nario" (for a given b) for probe 2, because we are assuming 
that it will be for sure systematically biased. A more bal- 
anced approach is to average the robustness along the direc- 
tion defined by the systematic bias vector b. This gives an 
"average robustness", which accounts for different possible 
sizes in the strength of the bia^. In order to perform the av- 
erage, we rotate the coordinate axes so that the new x-axis 
is aligned with the vector b (assuming here a 2-dimensional 
parameter space for simplicity): 



= Ab 



(28) 



where A is a suitable rotation matrix, and A' — A~^. Then 
the average robustness along the direction defined by b is 
given by 

Vy(x)-^da;= ^ / iy(x-)e"50ii- (29) 

where 

D = AF*A' (30) 
and W{x) is a suitable weighting function. A natural choice 

^ Notice that as we average R along b wc do not rc-evaluate 
the Fisher matrix of the probe as a function of b, but we simply 
translate the Fisher matrix found at the fiducial point (i.e., the 
true parameters values). The Fisher matrix typically depends only 
weakly on the fiducial model chosen, as long as we consider the 
models within the parameter confidence region. If the bias vector 
is not much larger than the statistical errors we can therefore 
approximate the Fisher matrix at the biased parameters values 
with the one evaluated at the fiducial point. 



for W is a Gaussian with characteristic scale for the bias 
given by the length of the bias vector, 



W{x) = 

so that Eq. (I29() becomes 
{R) = 



/2^\b\ 



(31) 



R,V2^\b\ 

(|F2||2Li+Sj)^/^ 
|FFi|i/2 



(32) 



-1/2 



The Gaussian weight is centered at the unbiased parame- 
ter values, but it also has a tail that stretches above the 
characteristic scale of the bias, in order to account for 
a potentially much larger bias. We have checked that the 
use of other weight functions (e.g., a top-hat weight out to 
a maximum bias value given by the size of the bias vector) 
give a qualitatively similar result. We define the quantity 
given by Eq. (|32p as the "average robustness" FoM. 

Finally, we can also combine the statistical and robust- 
ness FoMs to obtain an overall FoM expressing both the 
statistical power and the robustness to systematic bias of 
the probe as 



Tat = Rn X S, 
(T) = (R) X S, 



(33) 
(34) 



where S is given by Eq. ITU), while Rn and (R) by Eqs. (p7)l 
and (|29|l . respectively. 



3 PROPERTIES OF THE ROBUSTNESS FOM 

Before applying the above formalism to future dark energy 
probes, we wish to gain some further insight into the be- 
haviour of our robustness FoM by considering it in the con- 
text of a Gaussian toy model. We start with the normalized 
expression for the average robustness, Eq. (|32|l and assume 
now that the confidence regions of the two probes are identi- 
cal up to a roto-translation (and therefore the determinants 
of Fi , F2 are equal) . If moreover the prior is very weak we 
can approximate the posterior with the likelihood, hence 



(R)^2\^\\b\'D,, + ir/'. 



(35) 



Let us further assume that probes 1 and 2 are aligned, 
i.e., they have a degeneracy direction lying along the same 
straight line. This means also that their Fisher matrices are 
simultaneously diagonalizable (i.e. they commute) and that 
F is also diagonalizable. Since the bias vector b by definition 
connects the maximum likelihood points of the two probes, 
its direction is also aligned with one of the principal axis of 
the probes in this particular example. Then we can write 



D = A(F2 - F2F" V2)A" 
= F2^-F2°(F^'+F2°) 



(36) 
(37) 



where the superscript D denotes the diagonalized version of 
a matrix. The last step follows because for any matrix A 
diagonalized by A and any power k one has 



Ayl'=A~ 



(38) 
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Now let us denote the length of the j-th semiaxis of the i- 
th probe by tJij where (after diagonalization) the semiaxis 
j = 1(2) lies along the abscissa (ordinate) . Then we have 



0.33 S =1 .24 <R> =0.50 <T> =0.61 



Dll = 0-2.1 1 - 



2 



+ 4l 



and therefore 



(R) 



2(cr2,l(T2,2) 



(cr2 2 + cr2 



)l/2(,;,j2+ ^2 ^+^2^)1/2 



(39) 



(40) 



This expression shows that the average robustness is invari- 
ant with respect to rescaling of the axes: in fact, if the dis- 
tances along the abscissa, cri,i, (T2,i, |&|, are rescaled by an 
arbitrary factor, (R) does not change; the same applies in 
the y-direction. 

Since we assumed the ellipses to be congruent, we have 
two qualitatively different cases: orthogonal ellipses (_L), i.e. 
1^2,2 = cTi.i and (72,1 = cri,2; and parallel ellipses (||), i.e. 
(71,1 = (72,1 and cri_2 ~ (72,2. In the orthogonal case we obtain 



2r 



(7^(1 + ^2) 



-1/2 



(41) 



where r = (72,1/(72,2 measures the elongatedness of the el- 
lipses. In the parallel case we obtain instead for any r 



1 + 



-1/2 



(42) 



From these expressions we can derive some general conse- 
quences. Because of our choice of normalization, unbiased 
identical probes have unity robustness. In general, if the 
bias length is small with respect to the statistical errors of 
the second probe, then parallel probes are more robust than 
orthogonal ones. If the second probe is very elongated (de- 
generated) along the bias direction, i.e. r S> 1, then again 
parallel probes are more robust than orthogonal ones. If in- 
stead the degeneracy of the second probe lies orthogonally 
to the bias direction, r 1, there are two cases: paral- 
lel probes are more robust if the bias is smaller than the 
long axis (|6p < (72,2), but less robust in the opposite case. 
Of course the general case, with arbitrary bias direction and 
length and arbitrary sizes and orientation of the probes can- 
not be reduced to such simple conclusions. 

Armed with the above intuition, we now consider in 
Fig. [T] a numerical illustration of 4 different cases for the 
relative orientation of the two probes (orthogonal or parallel) 
and the direction of the bias vector (along the short or long 
semiaxis). The two sets of iso-likelihood contours enclose 
68% and 95% confidence levels; as above, the second probe 
(blue contours) has the same area as the first (i.e., Li = 
L2), but its degeneracy direction can be rotated, and its 
maximum likelihood value is displaced from the true value 
by a systematic bias (of fixed length in all cases), given by 
the green vector. The first probe (red contours) is assumed 
to be unbiased. The prior is 11 = diag(l, 1) (i.e. a prior of 1.0 
in each parameter with no correlations imposed). For each 
case, we give the corresponding statistical FoM, Eq. (|12p . 
the robustness FoMs, ^} and JglJ, and the total FoM (for 
the averaged robustness), Eq. (|34|) . 

The robustness FoM (with or without averaging) de- 
pends both on the direction along which the bias is directed 
and on the relative orientation of the degeneracy directions 




:0.34 S =1 .24 <R> =0.50 <T> =0.62 



0.59 S =1 .00 <R> =0.71 <T> =0.71 





Figure 1. Illustration of statistical and robustness FoM for a 
future probe (blue ellipses, 68% and 95% C.L.) whieh is system- 
atieally biased w.r.t. the present-day constraints (red ellipses) in 
the direction given by the green bias vector. The black dotted 
ellipses represent the combined constraints. Notice that the sta- 
tistical FoM (S) does not change in the presence of a systematic 
bias. 



of the two probes. When the bias is directed along the degen- 
eracy direction of probe 1 and probe 2 is aligned along that 
direction (lower left panel), the robustness is maximal. It de- 
creases if the two probes are orthogonal to each other, since 
this reduces the degree of overlap between them (upper pan- 
els). Finally, robustness is smallest when the two probes are 
aligned but the bias is direct orthogonally w.r.t the degener- 
acy direction (lower right panel), as argued above. Looking 
ahead to the application of the robustness formalism to the 
dark energy equation of state parameters in the next sec- 
tion, we can anticipate here that the most relevant case is 
the one where the two probes are similarly oriented (bot- 
tom panels of Fig. [T|. This is because different dark energy 
probes are typically degenerate in the equation of state pa- 
rameters along quite similar directions. Therefore, their rel- 
ative robustness can be expected to depend mainly on the 
orientation of the bias w.r.t. the main degeneracy direction. 

The statistical FoM is largest when the probes are or- 
thogonal to each other, as expected. Notice that the statis- 
tical FoM is unaffected by the bias, and only depends on 
the relative alignment of the two probes. For a given ori- 
entation and size of the bias vector, the total FoM allows 
one to decide which configuration for probe 2 is to be pre- 
ferred. For the example of Fig. [1] if the bias vector points 
along the degeneracy direction of probe 1 (left-hand side 
panels), one would prefer probe 2 to be aligned with probe 

1 ((T) = 0.71) as opposed to probe 2 being orthogonal to 
probe 1 ((T) — 0.61). If instead the bias is orthogonal to 
the degeneracy of probe 1 (right-hand side panels), then the 
best choice for probe 2 is for it to be orthogonal to probe 1 
((T) = 0.62 compared to (T) = 0.44). 

We can also ask what is the optimal orientation of probe 

2 with respect to probe 1 if one wanted to maximise its 
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robustness, given a bias direction. In Fig. [2l we plot both the 
statistical and the average robustness FoMs as a function of 
the rotation angle between the principal direction of the two 
probes. The average robustness is evaluated for 3 different 
directions of the bias (coloured vectors in the top panel) . We 
notice once more that the statistical FoM is maximised when 
the probes are orthogonal. However, the robustness FoM is 
maximised when the degeneracy direction of probe 2 is close 
to being aligned with the direction of the bias vector, as 
this maximizes the overlap with probe 1 even when probe 2 
suffers from a systematic error. Finally, increasing the length 
of the bias by a factor of 2 (fainter green vector in the top 
panel) reduces the overall average robustness. 

In summary, the robustness of a future probe is a func- 
tion of its statistical properties (i.e., the direction along 
which its main degeneracy is aligned, compared with the 
degeneracy direction of probe 1) as well as of the direction 
and size of the systematic bias. The performance of a future 
probe should be assessed by considering simultaneously its 
statistical power but also its robustness to systematics. Op- 
timizing a future dark energy experiment in terms of its sta- 
tistical errors alone would generically lead to an experiment 
which is less robust, for a given overall level of plausible sys- 
tematics. Any optimization procedure should therefore in- 
volve the complementary criteria of statistical strenght and 
robustness to systematic bias. 

We now turn to applying the above concept to the con- 
crete scenario of two classes of future dark energy missions, 
namely type la SN and BAO measurements. 



4 ROBUSTNESS OF FUTURE DARK ENERGY 
PROBES 

We consider a simple and widely used phenomenological de- 
scription of an evolving dark energy model, where the equa- 
tion of state is w^z) = wo -I- Waz/ (1 -I- z), characterized 
by th e two free par ameters {■WQ,'Wa) (|ChevalIier fc Polarskil 
I2OOII : iLindeij |2003| ). For probe 1 (representing current 
constraints on wo,Wa) we take a Gaussian approxima- 
tion to the joint likelihood resulting from the co mbina- 



tion o f Union 2 SNe la dat a (jAmanuUah et al.ll20ld l. SDSS 
BAO jPercival et al.ll2010l'). WMAP7 m easurements of the 
shift parameters ( Komatsu et al.ll2010l). and SHOES mea- 
surements of the Hubble constant ( Riess et al.ll2009l ). We 
further assume a flat Universe. For the prior on the dark 
energy parameters, we take a Gaussian centered at {wo + 
l,Wa) = (0,0) with Fisher matrix H = diag(l, 1/100). In 
the following, when we look at the Fisher matrix we mean 
the 2D marginalised Fisher matrix (i.e., marginalised down 
to the dark energy parameter space). Although in the rest 
of this paper we focus exclusively on the robustness FoM 
for dark energy parameters, we note that our robustness 
formalism is equally applicable to any other cosmological 
parameter one is interested in. 

In order to evaluate robustness, we need to specify the 
bias vector h. There are several plausible ways of doing this, 
and the outcome will depend on what one thinks a possible 
systematic bias might be due to. In our case, in order to illus- 
trate our new FoM, we determine h by assuming a possible 
systematic bias in the probe's observables, and then pro- 
jecting the resulting systematic shift onto the dark energy 




7t/4 7t/2 3Jt/4 

Relative angle between degeneracy directions 

Figure 2. Dependency of FoMs on the angle between the degen- 
eracy direction of the two probes. Upper panel: the red (blue) 
ellipses represent the 68% and 95% likelihood contours of probe 
1 (probe 2, which is potentially biased). The degeneracy direc- 
tion of probe 2 is offset by an angle d w.r.t probe 1. The three 
vectors gives three possible directions for the bias. Lower panel: 
value of statistical FoM S (black dashed line, right-hand axis), 
and average robustness FoM, {R), (coloured solid lines, left-hand 
axis, colour and thickness matching the bias vectors in the upper 
panel), as a function of the relative angle 0. Vertical coloured lines 
give the angle of each bias vector. 



parameter space of interest, as described in detail below. We 
stress that this is by no means the only procedure by which 
one can estimate b. Other assumptions about the origin of 
systematic errors will in general lead to a different b, and 
therefore to a different value for the robustness of the future 
probe. 

4.1 Future SN la Measurements 

We consider a survey dedicated to observing type la su- 
pernovae from space, with a redshift distribution like the 
one expected fro m SNAP, with 2000 SNe distributed as in 
iKim et al.ll|2004h . plus a low-z sample of 300 SNe distributed 
uniformly in the redshift range 0.03 < 2; < 0.08. The pro- 
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jected SNAP magnitude errors include both statistical and 
systematic components, and are modelled as follows: 



2 



0.15^ 



+ Ai. 



1 + Zb 

1 + 



(43) 



where Nb is the number of SNe in each bin centered at Zb and 
of width dz = 0.1. The second term on the right-hand side 
of Eq. H43p models a systematic floor that increases linearly 
with z up to a maximum of Aayst m ag per dz — 0.1 bin 
at Zmax = 1.7 (|Linder fc Hutereill2003l ). In order to evaluate 
the robustness of SNa data for different levels of systematics, 
we will consider values of ^syst — 0.01,0.02,0.05. 

We assume a flat universe with four parameters relevant 
for this analysis, matter density relative to critical JIm, equa- 
tion of state today wq, its variation with scale factor Wa, and 
a nuisance offset in the Hubble diagram M. Marginalizing 
over Sli\/ and A4 and assuming Agyst ~ 0.02, we find that 



our fiducial survey produces statistical errors of 



0.075 



and (Tiu^ = 0.30, corresponding to the black 68% C.L. ellipse 
in Fig. El 

The bias in the dark energy parameters, b, recon- 
structed from SN measurements induced by an arbitrary 
bias in the observed magnitudes 5m (z) can be derived 
from the Fisher matrix for SNe (e.g. Knox et al.l (|l998l ): 
iHuterer fc TurneJ (|200ll )'). and is given by 



dm{za 



dfij 



,iZa) 



(44) 



(45) 



,(>) 



where /it are the cosmological parameters, c, 
^^{F~^)ij{dm{za)/dfij) /a'^ and where a runs over redshift 
bins. We adopt a systematic bias of the same form as the 
"floor" that was previously included in the total statistical 
error: 



Smsysi^Za^ — ^syst 



1 + Zg 
1 + 



(46) 



Bias of this magnitude leads to the bias on cosmological 
parameters which can be calculated using Eqs. (|45|l and (|46p . 
and is shown as the blue curve in Fig. [3] Each point on 
the curve shows cumulative contributions to the excursion 
in Wo and Wa around the fiducial model (with {wo,Wa) — 
(—1,0)) for each of the 16 redshift bins we consider, z £ 
[0.1, 0.2], . . . , z G [1.6, 1.7]. In other words, points on the blue 
curve show cumulative contribution of the sum in Eq. (|45p . 

But this form of the bias assumes that the excursions in 
6m.{za) are of the same sign (taken to be positive), and equal 
to the maximum allowed value in Eq. (|46p . The worst-case 
bias to the dark energy parameters is obtained if 5m{za) 
changes sign in each redshift bin just so as to maximize 
the excursion in wq and Wa- Such a worst-case bias c an be 
straightforwardly calculated (|Huterer fc Taka"dall2005l ) 



bwt 
i 



|ci''| (5msys(2c«) 



(47) 



for a single dark energy parameter jii, where Smsys{za) > 
was taken by default. In other words, the systematic error 
takes a plus or minus sign equal to that of the coefficient 




-0.2 
-0.4 



Statistical 68% CL 

•— ■• Fiducial 5m(z) bias 

• — • Worst-case 6m(z) excursions 



-1.2 



-1.1 



-1 
W, 



-0.9 



-0.8 







Figure 3. Systematic bias in the WQ-Wa plane for future SNIa 
data. The square denotes our fiducial value and the ellipse gives 
the 68% CL statistical contour from future SNIa data. The blue 
curve shows the systematic bias given by Eq. I I46II . with points 
showing cumulative contributions from each of the 16 redshift 
bins - that is, cumulative value of the sum in Eq. l|45|I .For clarity, 
we explicitly label bias contributions accumulated by redshifts 
z = I and z = 1.7. The red segments denote the worst-case bias, 
where the sign of 5m{z) at each redshift bin conspires to shift the 
{wQjWa) value away from the true value ( "Maximum excursion 
bias"); see Eq. I I47I I. For clarity, we have also plotted the biases 
with the opposite sign relative to the fiducial model parameter 
values. 



Ca' in each redshift bin 0. Such a worst-case excursion in 
the {wo, Wa) plane is shown as the red curve with points in 
Fig. [3] We call this scenario the "maximum excursion bias" 
(MEB), and use it as an estimate for the bias vector b in the 
computation of our robustness FoM. 



4.2 Future Baryonic Acoustic Oscillations 
Measurements 

The second class of future probe we consider consists of 
a full-sky spectroscopic redshift survey modelling a future 
spacee mission with specifications close to WFIRST or Eu- 
clid (or a Stage-IV mission in the language of the DETF). 
The probe is fully specified by choosing a number of redshift 
bins and giving the expected number densities of galaxies 
per bin and the sky coverage, assumed here to be 20,000 
square degrees. Table [T] gives the redshift binning and the 
galaxy number densities, taken from the data published by 
the Euclid collaboration (jLaureiis et al.ll2009l 'l. We assume 
however that only half of these galaxies can be effectively 
employed (efficiency e = 0.5), corresponding to one half of 
values in the Table. 

In order to forecast the statistical errors on dark 



^ For multiple parameters, there is ambiguity to define the worst- 
case error, since a sign of Smsysiza) that makes excursion in uiq 
positive may actually make the Wa excursion negative or vice 
versa. We make a choice that the excursion in wq is positive in a 
given redshift bin, which determines the sign of (5maya(za); then 
the excursion in Wa in that bin is simply Cq™"' 5msys{za). 
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a{z) X 10"^ 



0.5-0.7 
0.7-0.9 
0.9-1.1 
1.1-1.3 
1.3-1.5 
1.5-1.7 
1.7-1.9 
1.9-2.1 



3.56 
2.42 
1.81 
1.44 
0.99 
0.55 
0.29 
0.15 



Table 1. Expected galaxy number densities per redshift bin in 
units of (/i/Mpc)"^ for the Euclid survey. 



ene rsy parameters, w e ado 
of ISeo fc EisensteirJ 



! adop t the I 
(|200a . l2007h 



the Fisher matrix method 

—^^y, also employed in 

lAmendola et al.l (|2005l ). Here we give a short summary of 
the method and refer to these papers for the implementa- 
tion details. In the limit where the survey volume 14urvcy 
is much larger than the scale of any features in Pobs(fc), it 
has been shown that the redshift survey Fisher matrix i n a 
redshift bin A2 can be approximated as (|Tegmarklll997l l 

91nPoba(fe,M) 91nPobs(fc,A*) 



2tvP dkd^ 
2(27r)3 



(48) 



Here, k, fi are the wavevector moduls and direction cosine 
with respect to the line of sight, respectively, and the deriva- 
tives are evaluated on the parameters fii of the fiducial 
model. The upper cut-off fcmax is chosen so as to avoid the 
non-linear regime, while the large-scale cut-off fcmin is set to 
O.OOlft/Mpc but its precise value has a very weak impact. 
VcH is the effective volume of the survey: 



Vcfi {k,fj,) = 



where Vau 



nePg{k,fi) 
ngPg{k,fj.) + 1 



Vsur 



(49) 



is the 20,000 square degrees survey volume 
contained in a given redshift bin. The galaxy comoving num- 
ber density rig (z) is assumed to be spatially constant within 
a redshift bin, while Pg is the galaxy spectrum defined be- 
low. The total Fisher matrix is obtained by summing over all 
the redshift bins of Table [1] The matter power spectrum in 
any given cosmology can be written in terms of the spectrum 
in the fiducial (or "reference" , subscript "ref ' ) cosmology as 



-fobs ( k, 



ref ^ ; "Tcf II 5 ' 



Pgik 



where 



Pg{krci±,hc!\\,z) = b{z)^ 



rcf 1^ 5 "Tcf II 5 



Vcf|| 



i-2 



rcf 



z)+PBhot , 

(50) 

-fmattcr(^5 ■S) 

(51) 

In Eq. I|50p . H{z) and D{z) are the Hubble parameter and 
the angular diameter distance, respectively, and the pref- 
actor encapsulates the geometrical distortions due to the 
Alcock-Paczynski effect (|Seo fc EisensteinI l2003l . 120071 '). k± 
and fcy are the wave-numbers across and along the line of 
sight in the given cosmology, and they are related to the 
wave-numbers calculated assuming the reference cosmology 
by fcrcfx = k±D{z) / D{z)rc{ and fc^ofn = k\\H{z)^ci/ H{z). 



Pshot is the unknown white shot noise that remains even 
after the conventional sh ot noise of inverse number den- 
sity has been subtracted (|Seo fc EisensteinI |2003| . l2007h . In 
Eq. H51|l . b{z) is the linear bias factor between galaxy and 
matter density distributions, /g(z) is the linear growth rate, 
I3{z) = fp (z)/b(z) is t he linear redshift-space distortion 



parameter (|Kaiseil Il987l ) and Pmattcr is the linear matter 



power spectrum. The fiducial values for the bias and the 
gr owth factor are b(z) = 1 and fg — Q'^j''^'' , respectively. 
In ldi Porto et all (|201ll ) it has been shown that the precise 
fiducial value of b{z) does not have a large impact on the 
results. 

This method employs all the information contained in 
the power spectrum, including the redshift distortion, not 
just the position of the baryonic wiggles. As above, we 
choose a flat fiducial cosmology with ~ 0.24, h = 0.7, 
Qde = 0.737, D.K = 0, ^Ibh'^ = 0.0223, = 0.96, 
uiQ = — 1, Wa = 0. Unlike in the SN case, we do not impose 
an explicit systematic floor in the forecasted BAO errors; 
the finite sky coverage and density of galaxies provide an ef- 
fective floor for any given BAO survey. As mentioned above, 
beside the cosmological parameters, for each redshift bin we 
also include as free parameters to be differentiated (and then 
marginalized) in the Fisher matrix a matter-galaxy bias fac- 
tor and an ad ditive shot noise term in the power spectrum 
(for details see lAmendola et al.l (|2005l l). These terms act as 
additional effective systematic floors. 

The systematic effect we assume for the redshift survey 
is a fractional error in estimating the value of the Hubble 
function H{zi) of magnitude A^yst = 0.001,0.002,0.005 in 
each bin i. Such a bias in H{z) propagates to a bias in the 
angular diameter distance D{z), as well, if the standard flat- 
space Friedman-Robertson- Walker relation 



D{z)^{l + z)-' 



dz' 
H{?) 



(52) 



holds true, which we assume here. The angular diameter 
distance bias is then related to the Hubble function bias by 



S{lnD) = -S{lnH) 



dz' 



{l + z)D{z)Jo m{z') 



H{z) 



(53) 



where we have used the assumption that the bias in In H is 
redshift-independent. This simple choice for modelling sys- 
tematic errors in BAO is meant to approximately capture a 
possible systematic shift in the baryon peak position due to 
e.g. the presence of isocurvature modes (|Zunckel et al.ll201Gl) 



or non-linear effects, of the kind described e.g. in Seo et akl 
(2008). A more realistic choice of systematic errors is difficult 
to model accurately (as, for example, a bias in H{z) and/or 
D{z) also modifies in general the whole spectrum behavior 
and the redshift distortions), and it is left for future work. 
Our present choice is meant as a simple illustration of the 
method and a first step towards evaluating the robustness 
FoM. 

If instead of the true matter power spectrum, P{k), 
we measure a spectrum that contains a systematic error 
5sa = S{\nHa) or 5sa — 5{\nDa) in the value of H{zc,) and 
D{za) (where the systematic shifts are related by Eq. ((53])), 
the maximum likelihood estimate for the i-th parameter will 
be shifted w.r.t. its true value by a bias given by (see e.g. 
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iTavlor et al.1 (|2007l )) 
1 



/ d^^k dk 



dlnPdlnP 



dfij dso 



Ss„ 



(54) 



(sum over repeated indexes). Analogously to the previous 
subsection we have defined 



1 f , ,2„ dlnPdlnP 
—— / dfik dk 



(55) 



In this particular case, however, the i-th parameters coin- 
cide with Ssi = S{\n Hi), 5 (In Di) and therefore the matrix 
Ca^ is the identity matrix. We can then directly project the 
systematic bias onto the dark energy parameters (wojWa), 
obtaining a bias vector b of the form 



13 



dwi 



d\nH{zp) 
H{zf,) dz' 



{l + zp)D{zp) 



dwi 



m dlnDizp) 



(56) 



5\nH{zi3) 



where 5\uH{zfi) = 0.001,0.002,0.005, the subscript /3 runs 
over the redshift bins, and I = Q,a. We have chosen to con- 
sider systematic shifts in the range of 0.1% to 0.5% to reflect 
ballpark estimates of what BAO systematic errors due e.g. 
to residual non-linear corrections might be. We stress once 
more that this is a simplified treatment used here mainly for 
illustration purposes of our method. 

We evaluate bi for each redshift bin and then estimate 
the maximum bias by following the same method discussed 
in the previous subsection. Here it happens that the contri- 
butions to bi are always positive and therefore automatically 
select the worst case scenario, i.e., the maximum excursion 
bias. The resulting maximum excursion bias for different 
levels of systematics is shown as the green vectors in the 
bottom panel of Fig. [l] together with the statistical errors 
from our BAO probe (blue ellipses) and current constraints 
(red ellipses), plotted for comparison. 



5 RESULTS 

Our results for the statistical and robustness FoMs are sum- 
marized in Table [2l where we give the values of our robust- 
ness, statistical and total FoM. We also show the value of 
the DETF FoM (normalized to the value obtained from the 
current probes) for comparison. 

First, by inspecting Fig. IH we notice that the system- 
atic bias projected onto the (wo,ifa) plane is much better 
aligned with the degeneracy direction of the probes for SNIa 
than for BAO. From our discussion in section |3] this leads to 
expect a higher value for the robustness FoM for SNIa than 
for BAO. Also, the size of the bias vectors in the dark en- 
ergy parameters is roughly comparable for SNIa and BAO, 
although in the latter case we have adopted a bias in the 
observables [H and D) which is a factor of 10 smaller than 
for the SNIa observables (the magnitudes) . Table [5] shows 
that indeed both the robustness and the average robustness 
FoMs are slightly larger for SNIa than for BAO across the 
range of systematic error levels we adopted for each probe. 
This is a consequence of the fact that the BAO bias leads 
to a smaller degree of overlap of the BAO constraints with 



the present-day constraints, which is a more serious lack of 
robustness than for the SNIa. In the latter case, although 
the bias vectors are slightly larger in the dark energy pa- 
rameters (typically by a factor of 2, cf Table [2]), the bias 
direction is well aligned with the statistical degeneracy, and 
therefore the reduction in the overlap between the present 
constraints and future SNIa constraints is less severe, trans- 
lating in a higher robustness. For the highest level of sys- 
tematic error in each case (0.5% for BAO and 5% for SNIa), 
we find that the robustness FoM for BAO is about a factor 
of 10 smaller than for SNIa. The average robustness of BAO 
is also smaller, but only by about 1/3, which refiects the 
more balanced assessment given by the average robustness. 
Thus, for our particular choice of systematics, our findings 
run against the general lore that BAO observations are more 
robust to systematics than SNIa. 

In terms of our statistical FoM, the SNIa survey is bet- 
ter by a factor of about 3, in good agreement with the result 
obtained from the usual DETF FoM. Taken together, the 
better values of both the statistical and robustness FoM for 
SNIa lead to a higher value of the total FoM for SNIa than 
for BAO. 

It is important to stress that our robustness results 
above are not a generic feature of SNIa and BAO obser- 
vations. Rather, they reflect our specific choices for the sys- 
tematic bias in the observables for BAO and SNIa. Other 
choices of systematic bias are possible and will in general 
give a different results for the robustness, which we shall 
explore in a dedicated paper. 



6 CONCLUSIONS 

We have introduced a new formalism to quantify the robust- 
ness of future dark energy probes to systematic bias, and ar- 
gued that this important new quantity should be taken into 
account when evaluating the performance of future surveys. 
In constrast to usual measures of statistical performance, 
our robustness FoMs depend on the direction and size of 
the systematic bias induced in the dark energy parameters 
by residual systematics in the observables. We have thus de- 
scribed an approach to include the effect of systematic errors 
in the dark energy figures of merit. 

We have applied this formalism to future SNIa and BAO 
probes by developping a simple phenomenological model of 
possible residual systematic errors. Our results indicate that 
- for the specific choice of systematics adopted here - SNIa 
are slightly more robust to systematics than BAO, despite 
having assumed a systematic shift in the observables for 
SNIa which is a factor of 10 larger than for BAO. Coupled 
with the higher statistical performance of SNIa, this would 
lead to prefer SNIa over BAO in terms of their overall FoM. 
It is clear however that this particular result cannot be gen- 
eralized beyond our choice of systematics and surveys. In 
a future work we will investigate how this result change by 
adopting more refined descriptions of the systematic bias for 
each probe. 
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Maximum excursion bias Maximum excursion bias 
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0.1% 


0.2% 


0.5% 


1% 


2% 


5% 


Robustness 


Rn 


Eq. 


C3 


1.4 


0.83 


0.026 


1.7 


1.3 


0.24 


Average robustness 


(R) 
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1.4 


1.1 


0.54 
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0.81 
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S 


Eq. 
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Total FoM 


Tn 
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1(33)1 
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11.9 
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11.9 
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0.36 
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0.34 
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Table 2. Robustness and statistical Figure of Merits for future BAO and SNIa surveys, for different levels of systematic errors in the 
observables. We also give the DETF FoM for comparison (normalized to its value from current constraints). We also give the length of 
the bias vector b in the (luoi'i'a) plane. The Maximum excursion bias errors refer to both D(zij) and H{zf,) in the case of BAO, and 
m{zi,) in the case of SNIa. 



SN 




Figure 4. Construction of the robustness FoM for a future SNIa 
survey (top panel) and a future BAO Euclid-like survey (bot- 
tom panel). Red ellipses show current 68%, 95% constraints (in 
a Gaussian approximation) from a combination of all available 
probes, blue ellipses show projected constraints from the future 
probe at the fiducial point (assumed to be ACDM). The green 
vectors show the systematic maximum excursion bias (MEB) for 
systematic erros of 1%, 2% and 5% for SNIa and 0.1%, 0.2% and 
0.5% for BAO. 
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APPENDIX A 



The generalization of Eq, 
probes proceeds as follows, 
always summarize current 
vations in one single joint 
from the combination of all 
(with Fisher matrix Fo). If 
probes, we can ask whether 
compatibl^ll- Eq. (|15|) gives 



pop to an arbitrary number of 
First, we notice that one can 
constraints from several obser- 
posterior. Let us call the data 
available present-day probes do 
one wishes to consider A'' future 
all of the A'' probes are mutually 
in this case 



9(djvdj\ 



■ di\do) 



n 

JV 

n 

j=2 



p{dj\d 



'i-i ■ 



• dido) 



p(dj |do) 



■ dido) 



p{dj\do) 



(58) 



(59) 



(60) 



where in the last line we have cancelled out the very last 
term in both the numerator and the denominator, so that 
the sum starts with j = 2. We now refer to Eq. (|17|l to 



^ An alternative test would be to check whether the A^-th probe 
is compatible with the previous N — 1 (assuming those are already 
available and they are free of systematics themselves) . In this case 
the relevant quantity is 

p(div|rfjv-i ■■■di) 



Ri\ 



(57) 



p(djv)p(djv_i ...di) 
which can be computed by appropriate substitutions in Eq. I I20II . 
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obtain 



p{dj\d 



'j-i ■ 



■ dido) 



.(j) 1-^012. ..(j-l) 
-0 



1/2 



(61) 



X exp 



1 ( *r 

2 (Mj-^'jMj 



+ M012...(j-l)-P()12...(j-l)Moi2...(3-l) 
- M012...j-Foi2...jM012...j)] , 

where the definitions correspond to those before, so that 
-Foi2...i = Fo + Yli^i^i, and in particular F012...N = F. 
Notice already that most terms in the numerator of Eq. (|60p 
will cancel. Similarly, following Eq. (|18() 



P(djMo) 



\Fo\'/' 
« \FoA'/' 



(2) 



(62) 



X exp 



Now one can evaluate Eq. (|60|l with the help of Eqs. ([6T 
and (I62J: 



TTT 11 i r. 1-1/0 xexp 



:(Moi-FbiMoi - 



In 



- (Af - l)/xoFo^o + ^ MojJoj^oj) 

We thus obtain for the robustness 
flail = i In |Fo,l - (iV - 1) In iFol - InF 

1 ^ _t _ 

~ n(^'Po^Fo^V:oi - (iV - l)AtoF()^o - m'-^M), 



(63) 



(64) 
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